How to remove cruft from your apache log

Keeping an eye on apache logs is an essential part of a webmaster’s job. Checking the logs help you detect performance problems and security issues. There are many log analyser software available to do this job effectively and create graphs, charts and other visualisations. Still, there are times when you need to check data directly from raw logs. But not all entries are useful. Access to a web page creates log entries not just for that page but for the JavaScript files, CSS files and images included in that page. This makes difficult to find information easily from raw logs. Separating such entries from main access log would help you to analyse data easily.

There are two ways to remove cruft from main log.

1. Conditional logging.
2. Separate (sub)domains for media and feeds.

Conditional logging

You can exclude some requests, by setting environment variables using SetEnvIf and adding env=!variable clause to the CustomLog directive in your apache configuration.

For example, if you have your images located in /images directory, you can exclude all requests to that directory from main log and if you want, log them in a separate log.

SetEnvIf Request_URI "^/images/" images
CustomLog /logs/example.com/access.log combined env=!images
CustomLog /logs/example.com/images_access.log combined env=images

See Conditional Logs in apache documentation.

This is what I do in my apache configuration:

SetEnvIf Request_URI "^/wordpress/" cruft
SetEnvIf Request_URI "^/robots\.txt$" cruft
SetEnvIf Request_URI "^/favicon\.ico$" cruft
CustomLog /logs/example.com/access.log combined env=!cruft

All my javascript, css, images and feeds are located in separate domains. See how to setup a separate sub-domain for media files in WordPress